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Abstract. The paper presents multi-sensor image fusion and its relevant 
framework and technical characteristics. The image fusion is divided into three 
level fusions: pixel level, feature level and decision level. It mainly discusses the 
image fusion algorithm at all levels of fusion, and then makes the summary and 
comparison of these algorithms. Since the high-level algorithms are related to 
some relevant practical applications of the image fusion, it is in general difficult 
to be summarized. So this paper also presents some typical algorithms of the fea- 
ture and decision levels from the perspective of the applications, to provide the 
necessary summary of the high level image fusion algorithm. Further, the three 
levels of implementation schemes are described, followed by the comparison and 
summary for the image evaluation criteria of the fusion method. Some problems 
and future directions about the multi-sensor image fusion are finally given. 


Keywords: Multi-sensor image fusion, image fusion classification, neural net- 
work image fusion, decision level fusion, fusion evaluation criteria. 


1 Introduction 


Multi-sensor image fusion is to adopt specific algorithms for combining two or more 
source images into new image information, which is more suitable for human visual 
perception, or for computer subsequent processing to meet the requirement of applica- 
tions, such as image segmentation, feature extraction, and target identification. 
Through the image fusion technology, we can effectively improve the reliability 
of the system and the utilization efficiency of image information. From the early 
1980s, the research of the multi-sensor image fusion has triggered widespread interest 
in the world, since it has a wide range of applications in automatic target recognition, 
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computer vision, remote sensing, robotics, medical image processing, military appli- 
cations and other fields [1-5], for example, infrared image and visible light image 
fusion can better help the helicopter pilot for navigation, and computed tomography 
(CT) and magnetic resonance imaging (MRI) are implemented to investigate the anat- 
omy and function of the body in both health and disease, which could provide reliable 
information for clinical diagnosis [3-6]. 

The multi-sensor image fusion is generally divided into three level fusions: the 
pixel level, the feature level and the decision level. Most of the image fusion algo- 
rithms are based on the pixel level fusion which is the main fusion algorithm, and the 
fusion algorithm on the feature and decision levels are much less mentioned in the 
literature. However, for the high-level image fusion including the feature and decision 
levels, the corresponding problems of the image fusion are more complex, and the 
fusion methods of the system are more ambiguous. For example, the feature level 
image fusion method is in general associated with specific applications, which has 
rich meanings, and can be used in various applications of the image processing [4-6]. 

In this paper, the multi-sensor image fusion is divided into three levels depending 
on our thoughts. The algorithm of the pixel level was presented in detail, and some 
typical image fusion algorithms in the feature and decision levels are reviewed from 
the view of application to provide the necessary summary of the high level image 
fusion algorithm. Specifically, for the pixel algorithms, the classification and compar- 
ison of the pixel level algorithm are in detail presented, followed by the summaries of 
the fusion algorithms on the feature level and decision level. 

This remainder of the paper is organized as follows. Section 2 outlines the prin- 
ciples of image fusion technology and architecture, technical features and image fu- 
sion classification methods, Section 3 presents a number of specific embodiments of 
the various methods and features, reviews some of the key issues of image fusion 
technology, and depicts some typical algorithms of the feature and the decision levels 
from the perspective of practical applications. Section 4 presents the image fusion 
evaluation criterion. The paper finally points out some problems and future directions 
about the multi-sensor image fusion. 


2 Theory and Characteristics of Image Fusion 


2.1 Image Fusion Classification 


The image fusion requires preliminary pretreatment, which includes several aspects: 
alignment, calibration, removing noise, etc. After completing the pretreatment, the 
image fusion can generally be divided into three levels: pixel level, feature level and 
decision level fusion, which is shown in Fig. 1. 

A. Pixel level fusion 

The pixel level fusion belongs to the lower level of fusion which directly fuses the 
image pixels information. Currently, most researches focus on this level. It fuses on 
the layer of the original image data collected from the imaging sensors. It is the high- 
est accuracy level data fusion in spite of its lowest level image fusion. It can provide 
the details are that the other levels do not have, but a large amount of information 
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Fig. 1. Three levels image fusion 


need be processed. The pixel level fusion called data level fusion is the basis of the 
high-level image fusion. 

B. Feature level fusion 

The feature level image fusion extracts feature information from each imaging sen- 
sor, and comprehensively analyzes and processes this information. During the fusion, 
the purpose of application and the scene of the multi-sensor image fusion are closely 
related to specific form and the feature information. Through the feature level image 
fusion, the related information is extracted from the original image, and it increases 
the credibility of the feature information, eliminates the false features, and creates a 
new composite feature, etc. The feature level image fusion is middle level for the 
preparation of the decision level fusion. Its advantage is of achieving considerable 
information compression, and being convenient for real-time processing. Since the 
extracted feature is directly related to decision analysis, the fusion result can furthest 
support to decision analysis in the high level fusion. 

C. Decision level fusion 

The decision level fusion can be simply described as: (1) classify and identify the 
feature information of each image, (2) form the corresponding results, (3) make the 
further fusion process. It is a higher level of information fusion, and the fusion results 
will provide the basis for controls or decisions. Therefore, the decision level fusion 
must combine with specific application and demand characteristics. Because the fea- 
ture directly affects the final result, we should selectively use the feature information 
to achieve the fusion purpose. The input information is all kinds of features, and the 
output results are about decision descriptions, so the decision level fusion is with the 
characteristics of the minimum data size and strong anti-jamming. 

Tab.1 shows the comparison of the different fusion levels performance characte- 
ristics. From the table and the previous description, we can see that the pixel level 
fusion is the most important and most fundamental multi-sensor image fusion 
methods, which can obtain a large amount of information and the best fusion 
performance [4-8]. 


Table 1. Comparison of image fusion levels and performance 


Features Levels Pixel level Feature level Decision level 

Information content maximum medium minimum 

Information loss minimum medium maximum 
Fault tolerance the worst medium the best 
Immunity the worst medium the best 
Dependence of the sensor maximum medium minimum 

Method difficulty hardest medium the easiest 

Pretreatment minimum medium maximum 
Classification performance the best medium the worst 
Open systems The worst medium the best 


2.2 Characteristics of Multi-sensor Image Fusion Technology 


The cost of the multi-sensor image fusion is less than that of single sensor, and it gets 
more or precise target information. The system has the following characteristics: 
(1) information redundancy: the confidence of the same scene target information may 
be different for the multi-sensor, but the image fusion will improve the overall target 
confidence. When a part of sensors are abnormal or damaged, it is also able to im- 
prove the system robustness (trustworthiness), (2) information complementary: 
the information complementary is achieved from the multi-sensor fusion to make 
the system getting scenery features, which is not able to be achieved using the 
single sensor (enlarge the spatial overlapping); (3) enlarge time overlapping: in a long 
time, the single sensor and the multi-sensors are capable of getting multiple informa- 
tion with a long time overlapping; (4) higher cost performance: as increasing of 
the number of sensors, the increasing cost of the system is less than its increasing 
information [1-8]. 


3 Image Fusion Methods 


3.1 Pixel Level Image Fusion 


Currently, the pixel level fusion methods have been more deeply studied. There are 
many kinds of image fusion algorithms, but the pixel level fusion can be broadly 
divided into two categories: (1) image fusion based on space domain and (2) image 
fusion based on transform domain. The significant differences between them are: 
the former directly fuses the pixels; the latter firstly transforms the image, and the 
object of fusion is then transformed coefficient instead of pixel values itself. The 
fusion coefficient may come from several or even all the original space pixel value 
which is obtained by the result of some transformation. The fusion process is shown 
in Fig.2. 
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Fig. 2. Pixel level image fusion flow 


3.1.1 Image fusion algorithm based on space domain 

1) Linear Weighting 

The linear weighting method is a simple image fusion method, which is to directly 
make the weighted stacking for corresponding pixels of the original images. If A; (i, j) 
represents the grey value of the k" image at the corresponding position (i, j), the fu- 
sion image B, (i, jJ) can be then obtained by the following formula: 


BG i)= MGA.) (1) 


DMA j)=1 (2) 


The advantages of the linear weighting method are that its concept is simple and it 
is suitable for real-time processing. The drawback is that the fusion image contains 
strong noise, especially, when the gray difference of the fusion image is significant, it 
may appear obvious splicing trace and poor visual effect. 

2) Component Substitution (CS) 

The component substitution algorithm is also called false color fusion algorithm. 
The most representative algorithm is the intensity-hue-saturation (IHS) transform, 
where ‘T’ represents density or brightness information, “H” represents chrominance 
information, and “S” represents saturation information. Although the IHS transform is 
very important false color fusion algorithm, it is just suitable for three band color 
image transformation. However, the current imaging systems, i.e. satellite imaging 
system, could provide far more than three bands. The latest IHS research, the related 
improvement and discussion to the spectral distortion can refer to the recent literature 
[2-4]. 

3) Probability Statistical Model Method 

There are many kinds of methods are based on probabilistic statistical model, such 
as Bayesian estimation theory, Markov random field theory [4-6], etc. The probability 
and statistics theory is needed to establish the corresponding optimization model, and 
the expectation maximization or simulated annealing algorithm can be used to solve 
the problem. 

3.1.2 Image fusion algorithm based on transform domain 

The transform domain image fusion algorithms include the Discrete Cosine Trans- 
form (DCT) and the Fast Fourier Transform (FFT), etc. However, the most important 
one is based on multi-scale and multi-resolution, such as Wavelet Transform (WT), 


Lifting Wavelet Transform (LWT), Discrete Wavelet Transform (DWT), Ridgelet 
Transform, Pyramid Decomposition, etc. Currently, this category obtains unprece- 
dented success in all kinds of the image fusion practical applications. The most prom- 
inent characteristics are that these algorithms have fast computing capabilities to satis- 
fy a large number of practical systems. 3.1.3 Other image fusion algorithm 

In recent years, there are some other image fusion algorithms in this field, such as 
Gradient-based multi-resolution image fusion method, which is based on the system 
transfer function theory and is established on basis of fusion decomposition model. It 
is significantly different from the multi-scale decomposition fusion algorithm. While 
also image fusion algorithm is based on ICA [7]; PDE-based approach to three- 
dimensional seismic image fusion algorithm [8]; total variation-based image fusion 
algorithm [9]; image fusion algorithm based on geometry [10]; and in recent years 
based on joint sparse representation theory and the theory of Compressive Sens- 
ing(CS) image fusion algorithm [11], [12]. These algorithms are based on broader 
basis theories, and they expand the meaning of image fusion itself. 


3.2 Feature Level Image Fusion 


The feature level algorithm is divided into two steps: target extraction and target feature 
fusion. In complex conditions, it is difficult to effectively extract the image target fea- 
ture, which is a hot research area in this field, so the feature level fusion is more depen- 
dent on the target feature whether to correctly obtain. When getting the correct target 
feature, all classification algorithms, such as support vector clustering, fuzzy clustering, 
and neural network algorithm, can be directly used to fuse the image feature. 


1) Joint Statistical 

When multi-sensor information is used for classification and decision, we need 
some type of discrimination scale, and need compare the sensed environment with the 
known feature. The joint statistics can be used to quickly and efficiently classify the 
unknown sample probability density function. 

2) Constrained Gauss-Markov Estimate 

The covariance matrix is used to storage and to restrain relevant information, and 
can be used as the fundamental geometric reasoning database. 

3) Extended Kalman Filter 

The Extended Kalman filter can be effectively adopted to make the image registra- 
tion and then implement the feature layer fusion. It also can reduce the uncertainty of 
the object location when appearing environment and sensor noise. 

4) Neural Network Image Fusion 

The characteristics of artificial neural network: inherent parallel structure and pa- 
rallel processing, distribution storage knowledge, fault tolerance, self-adaptability, 
etc. Network knowledge representation and acquisition process will be completed at 
the same time, so the execution speed can be accelerated. Additionally, the image 
noise exists inevitably in the real world, and even some information may be missing. 
In this case, the neural network fusion method can also reason in a reasonable way. 
After the neural network is trained, each image pixel is divided into several classes 


during image fusion, so that each image pixel has a membership function vector 
group. At present, most of the neural network is implemented by simulations, and 
digital signal processing chips and the relevant software are employed to simulate the 
parallel compute [13-17]. 


3.3 Decision Level Image Fusion 


The decision level fusion is a joint decision-making process and is more reliable than 
single sensor. Currently, the main problems of the decision level fusion: due to the 
time-varying characteristics of the environment and target, the prior knowledge is 
difficult to obtain. Simultaneously the enormous knowledge base and the design re- 
quirement of the object-oriented system hinder the theory and technology develop- 
ment of the decision level fusion. The main decision level image fusion methods are 
based on the cognitive model algorithm, which needs to use expert system and rela- 
tional database to analyze fusion judgment. The fusion algorithms on this level are 
mainly based on Bayes inference, evidence reasoning, neural networks, support vector 
machines and fuzzy integral fusion algorithm [7-11], etc. 


1) Bayesian estimation 

The bayesian estimation provides method which assembles sensor information in 
probability theory rule. The bayesian estimation is based on Bayes’ rule basal tech- 
nology. 

2) Fuzzy Logic 

The fuzzy logic is a class of multi-valued logic. The reasoning process uncertainty 
of the multi-sensor fusion can be directly represented by assigning real values for 
each proposition and operator from 0.0 to 1.0. The uncertainty model of the fusion 
process is then generated, as well as the subsequent consistency reasoning. 

3) Dempster-Shafer (D-S) 

The dempster-shafer (D-S) reasoning method: it is the bayesian approach 
extension, which uses probability interval and uncertainty interval to firm the 
likelihood function under the assumption of more evidence.The evidence theory can 
deal with uncertainty which is caused by unknown factors. Here, the belief function 
rather than probability is considered as a measure, bounding some event probability to 
establish trust function without explanation precise probability which is difficult to 
obtain. When constraints limit is strict probability, which becomes probabilistic 
theory. When evidence is relevant, we may consider using D-S theory promotion 
methods [18-21]. 


4 Performance Evaluation of Multi-sensor Image Fusion Method 


The performance evaluation of the image fusion effect is an important and meaningful 
research aspect, that is, how to evaluate the fusion effect or how to evaluate the image 
fusion quality. Although it is an important step in the image fusion, there still lack 
systematic evaluation methods. In the literature, the image fusion evaluation mainly 
consists of the two subjective and objective evaluation methods. 


4.1 Subjective Evaluation of Image Quality Fusion 


If a person is an image observer, the meaning of the image quality mainly includes 
two aspects: one is the image fidelity, and another is the image intelligibility. Over 
many years, people always want to seek quantitative measurement method about the 
image fidelity and intelligibility to be a criterion of image quality and image design 
system. Since a human visual system is not fully understood and a quantitative de- 
scription method about people's psychological factors is not generated, these issues 
should be resolved in the future. 


4.2 Objective Evaluation of Image Quality Fusion 


In order to overcome the human visual system, mental status, knowledge background 
and other uncertainty factors, we mostly use the objective evaluation criterion in the 
actual evaluation of the fusion effect. Objective evaluation determines and compares 
the pros and cons of various fusion methods through the calculation of fusion image 
statistical parameters [19-22]. 

In order to quantitatively evaluate the effect and quality of the fusion image, as- 
suming the two source images of participating into the fusion are A, B, its image size 
is MXN, and the fusion image F can be generated through the fusion treatment, sever- 
al parameters are defined as follows: 


1) Entropy 

Image entropy is an important index of image information richness, and the entro- 
py value shows how much the image average information is. The greater the entropy 
value of the fusion image is, the richer the information about the fusion image is, and 
the better the quality of the image fusion is. 

2) Average Gradient 

The average gradient can sensitively reflect the expression ability of image tiny de- 
tails contrasts. It can be not only used to evaluate image clarity, but also reflects the 
image tiny detail contrasts and texture transform features. 

3) Standard Deviation (SD) 

The SD reflects the discrete case of the image gray level relative to the average 
gray level, and it can be used to compare the size of image. If the SD is large, the 
image grayscale distribution is dispersed, and the image contrasts is big, so more 
information can be achieved. If the standard deviation and the image contrasts are 
small and the hue is homogeneous, we do not see too much information. 

4) Cross Entropy 

The cross entropy called the relative entropy is generally adopted to represent the 
difference between the two images. The smaller the cross-entropy is, the smaller the 
image difference is. 

5) Correlation Coefficient 

The correlation coefficient called similarity measure reflects the relevance of two 
images. After image fusion, it can reflect the spectrum information of the multispec- 
tral image to judge the retention capacity of the fused image spectrum. 


4.3 Combination of Subjective and Objective Methods 


The combination of subjective and objective quality evaluation method is an impor- 
tant index of the fusion image quality evaluation. It is the combination of artificial 
intelligence and expert systems fusion method to evaluate the effect [30]. This method 
does not require the ideal image, and does not take full account of the human visual 
characteristics. It can quantitatively evaluate the fusion image retaining much signifi- 
cant information (such as variance, contrast and edge information, etc.). It is a prac- 
tical, effective and versatile method of the image fusion quality evaluation. 


5 Conclusions 


In this paper, the multi-sensor image fusion is divided into three levels, which are 
made relatively systematic analyze and compare. This paper makes a systematic 
summary about the algorithms of the pixel, the feature and the decision levels from a 
large number of literatures. For lots of the pixel algorithms, the classification and 
comparison of the pixel level algorithm is presented in detail. Facing numerous the 
feature level and decision level algorithms, we list some typical algorithms for every- 
one's reference. 

The involved image fusion technology is very extensive, and it is restricted by time 
and conditions, so there is a lot of intensive detail work need to be continued in 
the future: 


1) Although there are many kinds of the image fusion methods, researchers do not 
think there is recognized standard and mature theory to support particular type image 
fusion. For the specific type image fusion, we need to develop corresponding unified 
mathematical model and integration framework. 

2) In this paper, we study the objective evaluation methods are just some common- 
ly evaluation methods, to avoid subjective evaluation of one-sidedness on some level. 
There are so many image fusion evaluation methods and standards, however, for the 
specific image fusion, the uniform evaluation criteria becomes an urgent problem to 
be studied. 

3) The rapid image fusion can be achieved by the optimization algorithm and re- 
ducing the algorithm complexity. The improvement of the fusion effect often sacrific- 
es the calculation speed and the algorithm complexity. The rapid and simple algo- 
rithm is the foundation of practical applications, so it is necessary to optimize the 
algorithm and study the fast algorithm. 
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